{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 观察Otto商品的特征进行PCA各维的方差，可以得到什么结论？（20分）\n",
    "PCA降维后数据量减小较多，原93维特征可用33个主成分表示；\n",
    "数量相对较少的主成分可以表示原始数据信息，的33个成分解释了原始特征85%的方差。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 对Otto商品tfidf特征，进行PCA降维，给出各维方差的分布图。（30分）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入工具包\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#读数据\n",
    "train = pd.read_csv(\"Otto_FE_train_tfidf.csv\")\n",
    "X_train = train.drop(['id', 'target'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PCA降维和方差\n",
    "pca = PCA(n_components=0.85, copy=True, whiten=False, svd_solver='auto')\n",
    "X_train_pca = pca.fit_transform(X_train)\n",
    "pca.n_components_ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca_test = PCA(n_components='mle', svd_solver='full')\n",
    "pca_test.fit(X_train)\n",
    "pca_test.n_components_ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAP4UlEQVR4nO3cf6zdd13H8efLlm0IcYOtGlyLLVk1lIAotSMRlTAdnUOKsYsdqDWZmSY0wQhiZ8KAislmDNXEmdjQah3qtsxfN664LAx/hODoHZMfZS5c6mTXLeyOluE0W+l4+8f5LpycnfZ+23tvb/s5z0dyc7/fz/fzPefzSXNfn08/53w/qSokSe36juVugCRpaRn0ktQ4g16SGmfQS1LjDHpJatzK5W7AqEsuuaTWrl273M2QpHPK/fff/0RVrRp37awL+rVr1zI9Pb3czZCkc0qS/zrRNZduJKlxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcWfdk7ELtXbnXc8re/imq5ehJZJ0dug1o0+yOclDSWaS7Bxz/ceTfCbJ8SRbR65tT/Kl7mf7YjVcktTPvEGfZAVwC3AVsAG4NsmGkWpfAX4Z+MuRe18KvB+4HNgEvD/JSxbebElSX31m9JuAmao6XFXHgNuALcMVqurhqvoc8K2Re98M3FNVR6rqKHAPsHkR2i1J6qlP0F8KPDJ0PtuV9dHr3iTXJ5lOMj03N9fzpSVJffQJ+owpq56v3+veqtpTVRurauOqVWO3U5YknaY+QT8LrBk6Xw082vP1F3KvJGkR9An6g8D6JOuSnAdsA6Z6vv7dwJVJXtJ9CHtlVyZJOkPmDfqqOg7sYBDQDwJ3VNWhJLuSvBUgyY8kmQWuAf4kyaHu3iPA7zAYLA4Cu7oySdIZ0uuBqao6ABwYKbtx6Pggg2WZcffuA/YtoI2SpAVwCwRJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrXK+iTbE7yUJKZJDvHXD8/ye3d9fuSrO3KX5Bkf5LPJ3kwyQ2L23xJ0nzmDfokK4BbgKuADcC1STaMVLsOOFpVlwG7gZu78muA86vq1cDrgF99bhCQJJ0ZfWb0m4CZqjpcVceA24AtI3W2APu74zuBK5IEKOBFSVYCLwSOAd9YlJZLknrpE/SXAo8Mnc92ZWPrVNVx4EngYgah/7/AY8BXgN+vqiOjb5Dk+iTTSabn5uZOuROSpBPrE/QZU1Y962wCngW+F1gHvDvJK55XsWpPVW2sqo2rVq3q0SRJUl99gn4WWDN0vhp49ER1umWaC4EjwNuBf6yqb1bV48AngY0LbbQkqb8+QX8QWJ9kXZLzgG3A1EidKWB7d7wVuLeqisFyzZsy8CLg9cB/LE7TJUl9zBv03Zr7DuBu4EHgjqo6lGRXkrd21fYCFyeZAX4DeO4rmLcALwa+wGDA+NOq+twi90GSdBIr+1SqqgPAgZGyG4eOn2bwVcrR+54aVy5JOnN8MlaSGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWpcr71uWrB2513PK3v4pquXoSWSdGY5o5ekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuN6BX2SzUkeSjKTZOeY6+cnub27fl+StUPXXpPkU0kOJfl8kgsWr/mSpPnMG/RJVgC3AFcBG4Brk2wYqXYdcLSqLgN2Azd3964EPgr8WlW9Cngj8M1Fa70kaV59ZvSbgJmqOlxVx4DbgC0jdbYA+7vjO4ErkgS4EvhcVX0WoKq+VlXPLk7TJUl99An6S4FHhs5nu7KxdarqOPAkcDHw/UAluTvJZ5K8d9wbJLk+yXSS6bm5uVPtgyTpJPoEfcaUVc86K4E3AO/ofv9skiueV7FqT1VtrKqNq1at6tEkSVJffYJ+FlgzdL4aePREdbp1+QuBI135P1fVE1X1f8AB4IcX2mhJUn99gv4gsD7JuiTnAduAqZE6U8D27ngrcG9VFXA38Jok39kNAD8BfHFxmi5J6mPlfBWq6niSHQxCewWwr6oOJdkFTFfVFLAXuDXJDIOZ/Lbu3qNJPsxgsCjgQFXdtUR9kSSNMW/QA1TVAQbLLsNlNw4dPw1cc4J7P8rgK5aSpGXgk7GS1LheM/qWrd35/JWkh2+6ehlaIklLwxm9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMZN/BYIJ+P2CJJa4Ixekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpce5eeRrc1VLSucQZvSQ1zqCXpMYZ9JLUOINekhrXK+iTbE7yUJKZJDvHXD8/ye3d9fuSrB25/vIkTyV5z+I0W5LU17xBn2QFcAtwFbABuDbJhpFq1wFHq+oyYDdw88j13cDHFt5cSdKp6jOj3wTMVNXhqjoG3AZsGamzBdjfHd8JXJEkAEneBhwGDi1OkyVJp6JP0F8KPDJ0PtuVja1TVceBJ4GLk7wI+C3ggwtvqiTpdPQJ+owpq551PgjsrqqnTvoGyfVJppNMz83N9WiSJKmvPk/GzgJrhs5XA4+eoM5skpXAhcAR4HJga5LfAy4CvpXk6ar6o+Gbq2oPsAdg48aNo4OIJGkB+gT9QWB9knXAfwPbgLeP1JkCtgOfArYC91ZVAT/2XIUkHwCeGg15SdLSmjfoq+p4kh3A3cAKYF9VHUqyC5iuqilgL3BrkhkGM/ltS9loSVJ/vTY1q6oDwIGRshuHjp8GrpnnNT5wGu2TJC2QT8ZKUuMMeklqnPvRLyL3qZd0NnJGL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOL91c4b4jRxJy8UZvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxPjC1zHyQStJSc0YvSY1zRn+WGjfTB2f7kk6dM3pJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNa7XpmZJNgN/CKwAPlJVN41cPx/4c+B1wNeAn6+qh5P8FHATcB5wDPjNqrp3Eds/kdzaWNKpmHdGn2QFcAtwFbABuDbJhpFq1wFHq+oyYDdwc1f+BPAzVfVqYDtw62I1XJLUT5+lm03ATFUdrqpjwG3AlpE6W4D93fGdwBVJUlUPVNWjXfkh4IJu9i9JOkP6BP2lwCND57Nd2dg6VXUceBK4eKTOzwEPVNUzo2+Q5Pok00mm5+bm+rZdktRDnzX6jCmrU6mT5FUMlnOuHPcGVbUH2AOwcePG0ddWT67dSxqnT9DPAmuGzlcDj56gzmySlcCFwBGAJKuBvwV+qaq+vOAW67Q4CEiTq8/SzUFgfZJ1Sc4DtgFTI3WmGHzYCrAVuLeqKslFwF3ADVX1ycVqtCSpv3ln9FV1PMkO4G4GX6/cV1WHkuwCpqtqCtgL3JpkhsFMflt3+w7gMuB9Sd7XlV1ZVY8vdkd0epzpS+3r9T36qjoAHBgpu3Ho+GngmjH3fQj40ALbKElagF5Br8njTF9qh1sgSFLjDHpJapxLNzplLutI5xZn9JLUOINekhrn0o0WjUs60tnJGb0kNc4ZvZbcuJk+ONuXzhRn9JLUOGf0Wlau60tLz6DXWelkA4CDg3RqDHo1wwFAGs+gV/P834EmnUEvjXGiAcCBQecig15aJA4COlsZ9NIScwDQcjPopWVysgfJHBy0mAx66RxyOh8sO2jIoJcm1GIOGg4mZzeDXtKSOp3BwYFjcRn0ks4ZDhqnx6CXNJEmadAw6CVpgU5l0DjZtaUaNNymWJIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4XkGfZHOSh5LMJNk55vr5SW7vrt+XZO3QtRu68oeSvHnxmi5J6mPeoE+yArgFuArYAFybZMNIteuAo1V1GbAbuLm7dwOwDXgVsBn44+71JElnSJ8Z/SZgpqoOV9Ux4DZgy0idLcD+7vhO4Iok6cpvq6pnquo/gZnu9SRJZ0iq6uQVkq3A5qr6le78F4HLq2rHUJ0vdHVmu/MvA5cDHwD+rao+2pXvBT5WVXeOvMf1wPXd6Q8ADy28a1wCPLEIr3OumuT+T3Lfwf5Pav+/r6pWjbvQZ1OzjCkbHR1OVKfPvVTVHmBPj7b0lmS6qjYu5mueSya5/5Pcd7D/k97/cfos3cwCa4bOVwOPnqhOkpXAhcCRnvdKkpZQn6A/CKxPsi7JeQw+XJ0aqTMFbO+OtwL31mBNaArY1n0rZx2wHvj04jRdktTHvEs3VXU8yQ7gbmAFsK+qDiXZBUxX1RSwF7g1yQyDmfy27t5DSe4AvggcB95ZVc8uUV9GLepS0Dlokvs/yX0H+z/p/X+eeT+MlSSd23wyVpIaZ9BLUuOaDPr5tmxoTZJ9SR7vnmd4ruylSe5J8qXu90uWs41LJcmaJJ9I8mCSQ0ne1ZVPSv8vSPLpJJ/t+v/Brnxdtx3Jl7rtSc5b7rYulSQrkjyQ5B+684npe1/NBX3PLRta82cMtpgYthP4eFWtBz7enbfoOPDuqnol8Hrgnd2/96T0/xngTVX1g8Brgc1JXs9gG5LdXf+PMtimpFXvAh4cOp+kvvfSXNDTb8uGplTVvzD4ttOw4W0p9gNvO6ONOkOq6rGq+kx3/D8M/uAvZXL6X1X1VHf6gu6ngDcx2I4EGu5/ktXA1cBHuvMwIX0/FS0G/aXAI0Pns13ZpPmeqnoMBmEIfPcyt2fJdbum/hBwHxPU/27p4t+Bx4F7gC8DX6+q412Vlv8G/gB4L/Ct7vxiJqfvvbUY9L22XVBbkrwY+Gvg16vqG8vdnjOpqp6tqtcyePJ8E/DKcdXObKuWXpK3AI9X1f3DxWOqNtf3U9Vnr5tzjdsuDHw1ycuq6rEkL2Mw22tSkhcwCPm/qKq/6Yonpv/PqaqvJ/knBp9VXJRkZTezbfVv4EeBtyb5aeAC4LsYzPAnoe+npMUZfZ8tGybB8LYU24G/X8a2LJluTXYv8GBVfXjo0qT0f1WSi7rjFwI/yeBzik8w2I4EGu1/Vd1QVaurai2Dv/N7q+odTEDfT1WTT8Z2I/wf8O0tG353mZu0pJL8FfBGBtuzfhV4P/B3wB3Ay4GvANdU1egHtue8JG8A/hX4PN9ep/1tBuv0k9D/1zD4wHEFg4nbHVW1K8krGHwR4aXAA8AvVNUzy9fSpZXkjcB7quotk9b3PpoMeknSt7W4dCNJGmLQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb9PyA10MOwhWNfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(pca.explained_variance_ratio_)), pca.explained_variance_ratio_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 采用train_test_split，从将数据集中随机抽取10000条记录（原始数据集太大，剩余数据抛弃，此部分SVM作业已经完成）。对这部分数据进行PCA降维，保留85%的能量。（20分）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 187)\n"
     ]
    }
   ],
   "source": [
    "#数据准备\n",
    "train_org = pd.read_csv(\"Otto_FE_train_org.csv\")\n",
    "train = train.drop([\"id\",\"target\"], axis=1)\n",
    "train1 =  pd.concat([train_org, train], axis = 1, ignore_index=False)\n",
    "del train_org\n",
    "del train\n",
    "#分开X和y\n",
    "y_train = train1['target'] \n",
    "X_train = train1.drop([\"target\"], axis=1)\n",
    "#随机选择样本10000\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train_part, X_val, y_train_part, y_val = train_test_split(X_train, y_train, train_size = 10000, random_state = 3)\n",
    "print (X_train_part.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#把小样本中的id去掉，id单独保存\n",
    "X_train_part_id=X_train_part[\"id\"]\n",
    "X_train_part=X_train_part.drop([\"id\"],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "主成分数量： 53\n"
     ]
    }
   ],
   "source": [
    "#PCA降维\n",
    "pca2 = PCA(n_components = 0.85)\n",
    "X_train_part_pca = pca2.fit_transform(X_train_part)\n",
    "print('主成分数量：', pca2.n_components_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#保存降维结果\n",
    "n_components = pca2.n_components_\n",
    "feat_names_pca = []\n",
    "for i in range(n_components):\n",
    "    feat_names_pca.append(\"pca_\" + str(i))\n",
    "    \n",
    "y = pd.Series(data =  y_train_part.values, name = 'target')\n",
    "id_=pd.Series(data= X_train_part_id.values,name = 'id')\n",
    "\n",
    "\n",
    "train_pca = pd.concat([id_, pd.DataFrame(columns = feat_names_pca, data = X_train_part_pca), y], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 对3中得到的数据（对降维后的数据），训练RBF核SVM，并对超参数（C和gamma）进行超参数调优。结果和用原始数据的情况比较（SVM部分作业结果）。（30分）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#准备数据\n",
    "x_trian_pca=train_pca.drop([\"id\",\"target\"], axis=1)\n",
    "y_trian_pca=train_pca['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7763\n",
      "{'C': 10, 'gamma': 1.0}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "#参数搜索范围\n",
    "Cs = [0.1, 1, 10, 100, 1000]\n",
    "gammas = np.logspace(-1, 1, 3)\n",
    "turned_parameters = dict(C=Cs, gamma=gammas)\n",
    "#生成学习器实例\n",
    "SVC1 = SVC(kernel='rbf', decision_function_shape='ovr', shrinking=True, random_state=0)\n",
    "#生成GridSearchCV实例\n",
    "grid = GridSearchCV(SVC1, turned_parameters, cv=3, scoring='accuracy', n_jobs=-1, return_train_score=True)\n",
    "#调用\n",
    "grid.fit(x_trian_pca, y_trian_pca)\n",
    "#输出最佳参数\n",
    "print(grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "                           decision_function_shape='ovr', degree=3,\n",
       "                           gamma='auto_deprecated', kernel='rbf', max_iter=-1,\n",
       "                           probability=False, random_state=0, shrinking=True,\n",
       "                           tol=0.001, verbose=False),\n",
       "             iid='warn', n_jobs=-1,\n",
       "             param_grid={'C': [0.1, 1, 10, 100, 1000],\n",
       "                         'gamma': array([ 0.1,  1. , 10. ])},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "             scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "SVC2 = SVC(kernel='rbf', decision_function_shape='ovr', shrinking=True, random_state=0)\n",
    "grid2 = GridSearchCV(SVC2, turned_parameters, cv=3, scoring='accuracy', n_jobs=-1, return_train_score=True)\n",
    "grid2.fit(X_train_part, y_train_part)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 10, 'gamma': 1.0}\n",
      "0.7806\n"
     ]
    }
   ],
   "source": [
    "print(grid2.best_params_)\n",
    "print(grid2.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面的结果对比可以看出，通过PCA降维后得到的结果（accuracy:0.7763）和利用原始数据得到的结果（accuracy:0.0.7806）差距不算大，降维后的数据还是能在很大程度上体现原始数据信息的。如果考虑训练时间的话，PCA降维后的数据优势还是非常大的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12.46852763  9.79045224 29.0780917   7.49936843  7.12302494 31.24752482\n",
      "  5.57227365  6.88125253 30.86921032  6.07173355  8.05602177 29.45230993\n",
      " 10.28295557  8.22600047 24.65218019]\n",
      "[36.55110097 29.75981196 83.3653187  19.49010634 20.99594768 86.26162259\n",
      " 14.57052922 21.55247506 88.95501884 15.72996004 22.80904762 88.27881591\n",
      " 19.94415776 23.39827975 77.65435259]\n"
     ]
    }
   ],
   "source": [
    "#训练时间\n",
    "fit_time_pca = grid.cv_results_['mean_fit_time']\n",
    "fit_time_org = grid2.cv_results_['mean_fit_time']\n",
    "print(fit_time_pca)\n",
    "print(fit_time_org)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
